3D model-based coding methods that need 3D reconstruction techniques are proposed for next-generation image coding methods. A method is presented that reconstructs 3D shapes of dynamic objects from image sequences captured using two cameras, thus avoiding the stereo correspondence problem. A coaxial camera system consisting of one moving and one static camera was developed. The optical axes of both cameras are precisely adjusted and have the same orientation using an optical system with true and half mirrors. The moving camera is moved along a straight horizontal line. This method can reconstruct 3D shapes of static objects as well as dynamic objects using motion vectors calculated from the moving camera images and revised using the static camera image. The method was tested successfully on real images by reconstructing a moving human shape.

This paper addresses the problem of estimating 3-D motion of a rigid object from a sequence of monocular 2-D images. The surface of object is assumed to be modeled with several patches, each of which is expressed by an implicit equation. The proposed method estimates the pose (i.e., the location and orientation) of object that corresponds to each image in the sequence: The sequence of the estimated poses gives the motion of the object. The estimation is done by solving a system of equations, each of which is typically an algebraic equation of low degree, that is derived from the expressions of the surface patches and image contours data: so the method does not require establishing the correspondence between successive two frames in the image sequence or computing optic flow. Allowing several-patch models for objects enables the proposed approach to deal with a great variety of objects. The paper includes a numerical example, where our aproach has been applied to a polyhedral object modeled with several patches.

A new application of the factorization method is reported for 3-D shape reconstruction from endoscope image sequences. The feasibility of the method is verified with some theoretical considerations and results of extensive experiments. This method was developed by Tomasi and Kanade, and improved by Poelman and Kanade, with the aim of achieving accurate shape reconstruction by using a large number of points and images, and robustly applying well-understood matrix computations. However, the latter stage of the method, called normalization, is not as easily understandable as the use of singular value decomposition in the first stage. In fact, as shown in this report, many choices are possible for this normalization and a variety of results have been obtained depending on the choice. This method is easy to understand, easy to implement, and provides sufficient accuracy when the approximation used for the optical system is reasonable. However, the details of the theoretical basis require further study.

Three dimensional model based coding methods are proposed as next generation image coding methods. These new representations need 3D reconstruction techniques. This paper presents a method that extracts the surfaces of static objects that occlude other objects from a spatiotemporal image captured with straight-line camera motion. We propose the concept of occlusion types and show that the occlusion types are restricted to only eight patterns. Furthermore, we show occlusion type pairs contain information that confirms the existence of surfaces. Occlusion information gives strong cues for segmentation and representation. The method can estimate not only the 3D positions of edge points but also the surfaces bounded by the edge points. We show that combinations of occlusion types contain information that can confirm surface existence. The method was tested successfully on real images by reconstructing flat and curved surfaces. Videos can be hierarchically structured with the method. The method makes various applications possible, such as object selective image communication and object selective video editing.

Understanding unknown objects in images is one of the most important fields of the computer vision. We are confronted with the problem of dealing with the ambiguity of the image information about unknown objects in the scene. The purpose of this paper is to propose a new object recognition method based on the fuzzy relation system and the fuzzy integral. In order to deal with the ambiguity of the image information, we apply the fuzzy theory to object recognition subjects. Firstly, we define the degree of similarity based on the fuzzy relation system among input images and object models. In the next, to avoid the uncertainty of relations between the input image and the 2-D aspects of models, we integrate the degree of similarity obtained from several input images by the fuzzy integral. This proposing method makes it possible to recognize the unknown objects correctly under the ambiguity of the image information. And the validity of our method is confirmed by the experiments with six kinds of chairs.

In this paper, we study three-dimensional motion estimation using optical flow. We construct a weighted quotient-form objective function that provides an unbiased estimator. Using this objective function with a certain projection operator as a weight drastically reduces the computational cost for estimation compared with using the maximum likelihood estimator. To reduce the variance of the estimator, we examine the weight, and we show by theoretical evaluations and simulations that, with an appropriate projection function, and when the noise variance is not too small, this objective function provides an estimator whose variance is smaller than that of the maximum likelihood estimator. The use of this projection is based on the knowledge that the depth function has a positive value (i. e., the object is in front of the camera) and that it is generally smooth.

This paper proposes a new method that can robustly recover 3D structure and 3D motion of 3D moving objects from a few multi-views. It recovers 3D feature points by obtaining intersections of back-projection lines which are connected from the camera's optical centers thorough projected feature points on the image planes corresponding to the different cameras. We show that our method needs only six views to suppress false 3D feature points in most cases by discussing the relation between the occurrence probability of false 3D feature points and the number of views. This discussion gives us a criterion to design the optimal multi-camera system for recovering 3D structure and 3D motion of 3D moving objects. An experimental multi-camera system is constructed to confirm the validity of our method. This system can take images from six different views at once and record motion image sequence from each view over a period of a few seconds. It is tested successfully on recovering the 3D structure of Vinus's plaster head and on recovering the 3D structure and 3D motion of a moving hand.

This paper presents a theoretically best algorithm within the framework of our image noise model for reconstructing 3-D from two views when all the feature points are on a planar surface. Pointing out that statistical bias is introduced if the least-squares scheme is used in the presence of image noise, we propose a scheme called renormalization, which automatically removes statistical bias. We also present an optimal correction scheme for canceling the effect of image noise in individual feature points. Finally, we show numerical simulation and confirm the effectiveness of our method.

We prove that the maximum likelihood estimator for estimating 3-D motion from noisy optical flow is not optimal", i.e., there is an unbiased estimator whose covariance matrix is smaller than that of the maximum likelihood estimator when a Gaussian noise distribution is assumed for a sufficiently large number of observed points. Since Gaussian assumption for the noise is given, the maximum likelihood estimator minimizes the mean square error of the observed optical flow. Though the maximum likehood estimator's covariance matrix usually reaches the Cramér-Rao lower bound in many statistical problems when the number of observed points is infinitely large, we show that the maximum likelihood estimator's covariance matrix does not reach the Cramér-Rao lower bound for the estimation of 3-D motion from noisy optical flow under such conditions. We formulate a superior estimator, whose covariance matrix is smaller than that of the maximum likelihood estimator, when the variance of the Gaussian noise is not very small.

This paper describes a noise resistant algorithm for estimating 3-D rigid motion from optical flow. We first discuss the problem of constructing the objective function to be minimized. If a Gaussian distribution is assumed for the niose, it is well-known that the least-squares minimization becomes the maximum likelihood estimation. However, the use of this objective function makes the minimization procedure more expensive because the program has to go through all the points in the image at each iteration. We therefore introduce an objective function that provides unbiased estimators. Using this function reduces computational costs. Furthermore, since good approximations can be analytically obtained for the function, using them as an initial guess we can apply an iterative minimization method to the function, which is expected to be stable. The effectiveness of this method is demonstrated by computer simulation.

In this paper, we first discuss on a framework for a 3D image display system which is the combination of passive sensing and active display technologies. The passive sensing enables to capture real scenes under natural condition. The active display enables to present arbitrary views with proper motion parallax following the observer's motion. The requirements of passive sensing technology for 3D image displays are discussed in comparison with those for robot vision. Then, a new stereo algorithm, called SEA (Stereo by Eye Array), which satisfies the requirements is described in detail. The SEA uses nine images captured by a 33 camera array. It has the following features for depth estimation: 1) Pixel-based correspondence search enables to obtain a dense and high-spatial-resolution depth map. 2) Correspondence ambiguity for linear edges with the orientation parallel to a particular baseline is eliminated by using multiple baselines with different orientations. 3) Occlusion can be easily detected and an occlusion-free depth map with sharp object boundaries is generated. The feasibility of the SEA is demonstrated by experiments by using real image data.

We can understand and recover a scene even from a picture or a line drawing. A number of methods have been developed for solving this problem. They have scarcely aimed to deal with scenes of multiple objects although they have ability to recognize three-dimensional shapes of every object. In this paper, challenging to solve this problem, we describe a method for deciding configurations of multiple objects. This method employs the assumption of coplanarity and the constraint of occlusion. The assumption of coplanarity generates the candidates of configurations of multiple objects and the constraint of occlusion prunes impossible configurations. By combining this method with a method of shape recovery for individual objects, we have implemented a system acquirig a three-dimensional information of scene including multiple objects from a monocular image.

This paper proposes a new method to determine the shape of a surface by learning the mapping between three image irradiances observed under illumination from three lighting directions and the corresponding surface gradient. The method uses Phong reflectance function to describe specular reflectance. Lambertian reflectance is included as a special case. A neural network is constructed to estimate the values of reflectance parameters and the object surface gradient distribution under the assumption that the values of reflectance parameters are not known in advance. The method reconstructs the surface gradient distribution after determining the values of reflectance parameters of a test object using two step neural network which consists of one to extract two gradient parameters from three image irradiances and its inverse one. The effectiveness of this proposed neural network is confirmed by computer simulations and by experiment with a real object.

A problem which often arises in computer vision is that of matching corresponding points of images. In the case of object recognition, for example, the computer compares new images to templates from a library of known objects. A common way to perform this comparison is to extract feature points from the images and compare these points with the template points. Another common example is the case of motion detection, where feature points of a video image are compared to those of the previous frame. Note that in both of these example, the point correspondence is complicated by the fact that the point sets are not only randomly ordered but have also been distorted by an unknown transformation and having quite different coordinates. In the case of object recognition, there exists a transformation from the object being viewed, to its projection onto the camera's imaging plane, while in the motion detection case, this transformation represents the motion (translation and rotation) of the ofject. If the parameters of the transformation are completely unknow, then all n! permutations must be compared (n : number of feature points). For each permutation, the ensuing transformation is computed using the least-squared projection method. The exponentially large computation required for this is prohibitive. A neural computational method is propopsed to solve these combinatorial problems. This method obtains the best correspondence matching and also finds the associated transform parameters. The method was applied to two dimensional point correspondence and three-to-two dimensional correspondence. Finally, this connectionist approach extends readily to a Boltzmann machine implementation. This implementation is desirable when the transformation is unknown, as it is less sensitive to local minima regardless of initial conditions.

This paper describes a new algorithm for finding the contours of a moving object in an image sequence. A distinctive feature of this algorithm is its complete bottom-up strategy from image data to a consistent contour description. In our algorithm, an input image sequence is immediately converted to a complete set of quasi logical spatio-temporal measures on each pixel, which provide constraints on varying brightness. Then, candidate regions in which to localize the contour are bounded based on consistent grouping among neighboring measures. This reduces drastically the ambiguity of contour location. Finally, Some mid-level constraints on spatial and temporal smoothness of moving boundaries are invoked, and they are combined with these low-level measures in the candidate regions. This is performed efficiently by the regularization over the restricted trajectory of the moving boundary in the candidate regions. Since any quantity is dimensionless, the results are not affected by varying conditions of camera and objects. We examine the efficiency of this algorithm through several experiments on real NTSC motion pictures with dynamic background and natulal textures.

This paper describes a noise resistant algorithm for recovering the three-dimensional motion of a rigid object from optical flow. First, it is shown that in the absence of noise three-demensional motion can be obtained exactly by a linear algorithm except in the special case in which the surface of the object is on a general quadratic surface passing through the viewpoint, and the normal vector of the surface at the viewpoint is perpendicular to the translation velocity vector. In the presence of noise, an evaluation function is introduced based on the least squares method. It is shown, however, that the solution which minimizes the evaluation function is not always optimal due to statistical bias. To deal with this problem, a method to eliminate the statistical bias in the evaluation function is proposed for zero mean white noise. Once the statistical bias is eliminated, the solution of the linear algorithm coincides with the correct solution by means of expectation. In this linear algorithm, only the eigenvector corresponding to the zero eigenvalue of a 33 matrix is necessary to find the translational velocity. Once the translational velocity is obtained, the rotational velocity can be computed directly. This method is also shown to be noise resistant by computer simulation.

In this paper, we define an attributed random graph, which can be considered as a generalization of conventional ones, to include multiple attributes as well as numeric attribute instead of a single nominal attribute in random vertices and edges. Then we derive the probability equations for an attributed graph to be an outcome graph of the attributed random graph, and the equations for the entropy calculation of the attributed random graph. Finally, we propose the application areas to computer vision and machine learning using these concepts.

We describe an approach for modelling a person's face for model-based coding. The goal is to estimate the 3D shape by combining the contour analysis and shading analysis of the human face image in order to increase the quality of the estimated 3D shape. The motivation for combining contour and shading cues comes from the observation that the shading cue leads to severe errors near the occluding boundary, while the occluding contour cue provides incomplete surface information in regions away from contours. Towards this, we use the deformable model as the common level of integration such that a higher-quality measurement will dominate the depth estimate. The feasibility of our approach is demonstrated using a real facial image.

We describe an approach to describe moving pictures in terms of their structural properties for video editing, video indexing, and video coding. The description contains 2D shape, motion, spatial relation, and relative depth of each region. To obtain the description, we develop the incremental segmentation scheme which includes dynamic occlusion analysis to determine relative depths of several objects. The scheme has been designed along the analysis-by-synthesis" approach, and uses a sequence of images to estimate object boundaries and motion information successively/incrementally. The scheme consists of three components: motion estimation, prediction with dynamic occlusion analysis, and update of the segmentation results. By combining the information from extended (longer) image sequences, and also by treating the segmentation and dynamic occlusion analysis simultaneously, the scheme attempts to improve successively over time the accuracy of the object boundary and motion estimation.

In this paper we propose a new application of Wu's mechanical theorem proving method to reconstruct polyhedra in 3-D space from their projection image. First we set up three groups of equations. The first group is of the geometric relations expressing that vertices are on a plane segment, on a line segment, and forming angle in 3-D space. The second is of those relations on image plane. And the rest is of the relations between the vertices in 3-D space and their correspondence on image plane. Next, we classify all the groups of equations into two sets, a set of hypotheses and a conjecture. We apply this method to seven cases of models. Then, we apply Wu's method to prove that the hypotheses follow the conjecture and obtain pseudodivided remainders of the conjectures, which represent relations of angles or lengths between 3-D space and their projected image. By this method we obtained new geometrical relations for seven cases of models. We also show that, in the region in image plane where corresponding spatial measures cannot reconstructed, leading coefficients of hypotheses polynomials approach to zero. If the vertex of an image angle is in such regions, we cannot calculate its spatial angle by direct manipulation of the hypothesis polynomials and the conjecture polynomial. But we show that by stability analysis of the pseudodivided remainder the spatial angles can be calculated even in those regions.